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The  Case  For  Improved  Methods 


Arms  Fiascoes  Lead  to  Alarm  Inside  Pentagon 

-  New  York  Times,  8  June  2005 


80+  current  weapons  development  projects 
Totalling  $1.47  trillion,  $300B  over  budget 


(The  services)  "push  the  technology 
beyond  what  a  contractor  is  capabie  of 
achieving,"  said  (a  former  weapons- 
buying  official) 
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Challenge: 

Costing  Immature  Projects 


Concept 

Refinement 

Concept 
./  Chfclston 

Technology 

Development 

System  Development 
&  Demonstration 
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Source:  Technology  Readiness  Level  Calculator,  Assessing  Technology  Readiness 

&  Development  Seminar,  William  L.  Nolte,  28  April  2005 


Sy&lAm  Launch 
A  Operalicns 


TftL9 


Actual  system  “flight  proven”  through  successful 
mission  operations 


Problem: 

Low-TRL 

Technologies 


SyalemJSub&ystBm 


Tectinology 

Demonstration 


Actual  system  completed  and  "flight  qualified” 
through  test  and  demonstration  (Ground  or  Flight) 

System  prototype  demonstration  in  a  space 
environment 

System/subsystem  model  or  prototype  demonstratEon 
in  a  relevant  environment  (Ground  or  Space) 

Component  andyor  breadboard  validation  in  relevant 
environment 

Component  andVor  breadboard  validation  in  laboratory 
environment 

^Analytical  and  experimental  critical  function  ar>d/or 
icharacteristicproof-of-concept 

=Techno]ogy  concept  and/or  application  formulated 


:Basic  principles  observed  and  reported 
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Special  Class  of  Projects: 
The  “Nasty”  Ones 


Cost 


Technology 
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The  Trillion  $  Question 


If  new  technologies  are 
ventures  into  the  unknown 
then  how  can  their  cost  of 
development  be  obtained? 


Share  of 
Estimate 
Explainable 
By... 


A 


Conventional 


Alternative 

Methods... 


Costing  Methods 


-  100% 
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Distance  Into  Future 
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Answer;  Don’t  Trust  Any 
One,  or  Two,  or  Three  Methods 


Candidate  Estimates 


Combining  multiple  estimating  methods: 

•  Balances  strengths  and  weaknesses. 

•  Provides  cross  validation. 

•  Produces  a  range. 
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Support  for  the 
Meta-Estimating  Approach 


9  Reduced  failure  due  to  data  /  knowledge  scarcity  -  candidate 
estimating  methods  have  purposefully  unalike  data  requirements. 

o  Reduced  risk  -  estimating  methods  fare  differently  depending  on  the 
scenario. 

9  Reduced  bias  -  estimating  methods  based  on  alike  data  are  more 
likely  to  yield  the  same  result.  If  it  is  a  systemically  errant  result,  then 
it  is  more  likely  to  be  accepted.  Using  unalike  data  lessens  the  chance 
of  this  error. 

9  More  robust  -  not  all  estimating  methods  need  be  used  every  time. 
Those  used  would  depend  on  data  availability. 

9  Built-in  vaiidation  -  when  very  different  methods  agree,  there  is 
better  support  for  an  estimate. 
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Candidate  Estimating  Methods  for 
Advanced  Technology  Development 


Method 

Summary 

Data  Required 

Strengths 

Potential  Weakness 

Expert  Opinion 
via 

Muitidimensionai 

Delphi 

Combines  a  classic 
consensus-building 
process  with  a  method 
for  improved  accuracy 
in  comparative 
cognitive  exercises. 

•  Sufficient  numbers  of 
‘experts’  within  a  given 
domain 

•  Knowledge  of  overall 
costs  for  similar 
research  programs 

•  Accuracy  proven 
in  many  different 
applications 

•  Works  well  with 
limited  data 

•  Shortages  of  qualified 
‘experts’ 

•  Experts’  ignorance  of 
the  true  cost  of 
research 

Model  Derived 

From  Past 
Experience 

A  parametric  model 
developed  from  a 
record  of  past  projects 
undertaken  at  research 
labs. 

•  Multiyear  budget  data 
on  past  research 
programs 

•  Some  descriptive 
information  about  these 
programs 

•  Would  result  in 
an  extremely 
easy  to  use 
parametric  model 

•  Information  on 
research  funding  may 
be  difficult  to  obtain 

Financial 

Forensics 

Isolates  basic  R&D 
costs  as  a  component 
of  product  net  revenue. 

•  Breakdown  of  company 
cost  structure 

•  Share  of  firm  R&D 
attributable  to  specific 
technologies 

•  May  allow 
recovery  of  R&D 
expense  for  a 
wide  range  of 
products,  firms 

•  Inability  to  trace  back 
to  firms’  R&D  costs 

•  Past  R&D  expense 
may  not  be  indicative 
of  the  future 

Continuing  Cost 
of  Research 

Extrapolates  future 

R&D  expense  given 
funding  levels  to-date. 

•  Knowledge  of  ongoing 
costs  for  related 
research 

•  Accurate  given 
steady  cost  of 
research  over  a 
known  future 
duration 

•  Past  R&D  expense 
may  not  be  indicative 
of  the  future 

•  Ignorance  of  true 
research  costs 
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Caution:  Some  Problems  Are  Just  Hard, 

Not  Cutting  Edge 


Tangled  Ball  of  Simple  Simple  Ball  of  Weird  Stuff  - 
Stuff  -  Complex  Problem  Cutting  Edge  Technology 
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Expert  Opinion  Via 
Muitidimensionai  Delphi 


Problem:  New  technologies’  cost  is  hard  for  even  experts  to  grasp. 
Solution:  Mate  expert  opinion  with  proven,  intuition-based  methods. 

In  this  approach: 

o  Experts  are  recruited  with  knowiedge  of  a  variety  of  technicai  projects. 

o  They  are  then  given  ‘reference’  projects  with  known  cost  and  unknown 
projects  that  need  to  be  estimated. 

o  The  experts  are  asked  to  compare  aii  projects,  reference  vs.  unknown, 
and  so  on. 

o  After  this  they  are  shown  each  others’  comparisons  and  rationaies. 
These  other  comparisons  will  provoke  further  reflection  and  encourage 
people  to  further  refine  their  choices. 

o  After  2  or  3  rounds,  a  set  of  consensus  comparisons  will  exist  and  then 
be  input  to  the  paired  comparisons  algorithm,  to  provide  estimates. 
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Expert  Comparative  Assessment  Input  to 
AccuScope  Paired  Comparisons  Tool 


Expert  1 :  “I  think  A  is  4  times  as  big  as  B.” 

Expert  2:  “I  think  A  is  3  times  as  big  as  B.” 


Expert  3:  “I  think  A  is  4.5  times  as  big  as  B.” 

»>  Combined  assessment:  “A  is  3.8  times  as  big  as  B.” 
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The  ‘Continuing  Cost  of  Research’ 

Method  ^ 


Project  future  R&D 
expenses  over 
period  expected  to 
still  be  needed  for 
a  technology’s 
evolution. 


Research  costs 

over  time  trl  7 


Extrapolating  from 
recent  past 
experience. 


□fferent  shades  represent  different  cost  components 


TRL3  time 


Already  the  default  method 
of  R&D  costing? 
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Two  Questions  To  Answer  For  the 
‘Continuing  Cost  of  Research’  Method 


A.  “How  many  years  till  TRL  7?”  Can  be  answered  by  estimating: 

•  The  number  of  years  required  for  previous  generational  evolutions 
of  the  target  technology 

•  Whether  future  evolutions  will  require  a  similar  number  of  years. 

It  is  particularly  helpful  if  the  rate  of  change  in  the  technology  can 
be  inferred. 

B.  “How  much  is  spent  today  on  yearly  R&D?”  Historical  costs  can 
be  recovered  using  whatever  records  necessary,  while  future  costs 
can  be  determined  by  estimating  whether  costs  will  remain  stable  or 
change  due  to  labor,  capital  equipment,  test,  prototype  or  other  needs. 

The  cost  of  technology  progression  can  be  obtained  by  multiplying 
(A)  and  (B),  probably  best  done  on  a  yearly  or  lesser  basis  so 
that  varying  funding  requirements  can  be  captured. 
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Caution:  Technology  Forecasting  is 

Fraught  With  Error 


Maturation 

Curve 
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Visualization  Methods  For  Determining 
Technological  Maturity 


“How  related  are  topics?” 


“How  are 
patents 
related?” 


* 


Ccrhri 

Civn 

9l«ir#| 


r*inr4l 


“What  is  the  relation  between 
inventions  and  companies?” 


What  is 
the 
patent 
rate?” 
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The  ‘Model  Derived  From  Past 
Experience’  Method 

The  model  would  be  based  on  past  experience  conducting  research 
projects.  Questions  to  ask  when  obtaining  data: 

o  How  successful  was  the  past  outcome? 
o  Over  how  many  years  was  research  conducted? 
o  How  much  was  spent  each  year? 
d  How  many  staff  were  involved  each  year? 
o  What  was  the  budget  profile? 
o  How  novel  was  the  technology? 
o  Were  there  stoppages  in  support? 
o  How  volatile  were  requirements  for  this  technology? 
o  What  is  the  state  of  any  industry  producing  something  similar? 
o  What  was  the  difficulty  rating  for  the  technology? 
o  How  was  research  organized,  as  a  separate  lab,  multiple  teams,  etc? 
o  Etc! 
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The  ‘Financial  Forensics’  Method 


How  much  has  it  cost  someone  else? 

a  Profit-seeking  firms  price  products  so  sunk  costs  can  be 
recovered  after  an  aliowance  for  profit  and  other  expenses. 
Some  of  those  sunk  costs  are  for  R&D. 

a  Find  firms’  R&D  expenses  for  generations  of  products  by 
adopting  competitive  inteiliqence  techniques. 

Exampies: 

a  A  private  rocket  design  &  development  costs  $20-40M  each 
a  New  drugs  cost  about  $897M  each. 
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Combining  Estimates: 
A  Nuanced  Approach 


o  Can  the  past  indicate  anything  about  the 
future  or  does  cutting  edge  technology 
development  operate  according  to  an 
entirely  dissimi  ar  production  function? 

o  How  can  technologies  be  isolated  for 
analysis,  if  need  be,  from  the  systems  into 
which  they  are  integrated? 

o  How  can  a  “technology’s  readiness”  level 
be  precisely  described,  so  that  the 
transition  between  earlier  and  later  stages  is 
correctly  gauged? 


Candidate  Estimates 


nent  after  adjusti 


_ ►$ 

-rr 

Adjustments 

Based 

On  Out-of-Model 
Knowledge 

o  If  necessary,  how  can  technology  improvements  be  normalized  so  that 
qualitative  changes  are  differentiated  from  quantitative  ones? 

o  Can  lessons  learned  from  one  technology,  such  as  civilian  solar  cells,  be 
appiied  to  another,  such  as  spacecraft  solar  arrays? 

o  In  order  to  bring  about  technology  innovation,  what  balance  of  inputs  to 
the  “R&D  production  function”  is  reguired,  including  labor,  capital 
equipment  and  dispensable  material?  Does  this  balance  change  as  a 
technology  matures? 

o  How  can  market  developments  (serendipitous  discoveries,  etc.)  be 
controlled  for  to  generalize  lessons  learned  from  a  specific  technology’s 
evolution? 
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Analysis  Considerations 


o  How  can  technologies  be  isolated  for  analysis,  if  need  be,  from 
the  systems  into  which  they  are  integrated? 

o  How  can  a  technology’s  “readiness  level”  be  precisely 
described,  so  that  the  transition  between  earlier  and  later 
stages  is  correctly  gauged? 

o  If  necessary,  how  can  technology  improvements  be  normalized 
so  that  qualitative  changes  are  differentiated  from  quantitative 
ones? 

o  Can  lessons  learned  from  one  technology,  such  as  civilian  solar 
cells,  be  applied  to  another  such  as  spacecraft  solar  arrays? 
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Evolution  of  Technology  Considerations 


o  Do  technologies  bear  any  resemblance  to  one  another  in 
their  aspects  of  development?  Or. . . 

o  Does  cutting  edge  technology  development  operate  according 

to  an  entirely  dissimilar  production  function? 

o  Is  there  a  difference  between  technologies  whose  “time  has 
come”  through  continuous  development  vs.  those  that 
serendipitouslv  arise  “out  of  the  clear  blue  sky”? 

o  Do  technology  adoption  ‘curves’  follow  regular  and 
repeated  profiles?  Is  the  shape  of  these  profiles  consistent 
across  generations? 
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Market  Considerations 


o  How  does  cooperation  with  suppliers  or  competitors  sway  development? 

o  Do  venture  capital  flows  indicate  technologies  that  are  about  to  mature,  or 
do  they  in  fact  spur  maturation? 

o  How  do  market  characteristics,  and  the  broader  environment  in  which  firms 
innovate,  affect  technological  development? 

o  How  can  market  developments  be  controlled  for  in  generalizing  the  lessons 
learned  from  a  specific  technology’s  evolution? 

o  Do  technology  adoption  ‘curves’  follow  regular  and  repeated  profiles?  Is 

the  shape  of  these  profiles  consistent  across  generations? 

o  All  other  factors  aside,  do  differing  productivity  levels  within  firms  persist, 
so  that  a  company  which  has  innovated  well  in  the  past  will  continue  to  do  so 
in  the  future?  What  factors  lead  to  this  persistence? 

o  What  impact  do  potential  commercial  spin-offs  have  on  a  technology? 
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Intra-Organizational  Considerations 


o  What  is  the  difference  between  pure  and  directed  R&D?  Do 
firms,  over  time,  efficiently  internalize  their  pure  R&D  costs  into 
product  pricing? 

9  What  is  the  impact  between  R&D  conducted  entirely  within  one 
laboratory  or  shared  between  many  external  participants? 

9  Does  innovation  experience  returns  to  scale?  For  instance, 
does  a  dollar  spent  at  an  early  readiness  level  earn  the  same 
improvement  as  a  dollar  spent  later  on? 

9  In  order  to  bring  about  technology  innovation,  what  balance  of 
inputs  to  the  “R&D  production  function”  is  required,  including 
labor,  capital  equipment  and  dispensable  material?  Does  this 
balance  change  as  a  technology  matures? 
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Advanced  Estimating  Methods  in  Context 
With  Conventionai  Approaches 
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Stable  Technologies 
Estimated 

Conventionally  Using 
The  (Integrated)  SEER-H 
Costing  Model 


Electro-Optical 
Component  Estimated 
Using  SEER-H’s 
SpyGlass  Model 


Advanced 
Technologies 
Estimated 
Using  Methods 
Outiined  Here 


Balance  of  Future 
System  Estimated 
Using  Far  Out  Model 
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Who  Are  These  Scientists 
and  Innovators? 


Lee  at  galorath.eom 
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Who  Are  These  Scientists 
and  Innovators? 


Ada  Lovelace 


Hopper 


Lee  at  galorath.eom 
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